Decoding Nikon NEF Files

NEF is the file format Nikon cameras use to store RAW pictures. There is no actionable documentation on the format. This page collects what I have been piecing together from various sources and by reading the source code of dcraw↗.

All of this was verified on images taken with my Nikon D40 and on sample NEF images I have found online↗.

While it pretends to be a fancy special file format, in reality NEF files are mostly just TIFF with some custom tags. If you want to decode NEF, you should read up o TIFF first; You should know what IFDs and sub-IFDs are and how to access them, what dir-entry tags and types are and how to read them, how strips work, and so on.

The special tags Nikon added to TIFF are known↗.

General Structure

NEF files contain one main IFD. It has an uncompressed RGB thumbnail of the image.

The direntry with tag 0x14A "SubIfd" of the main IFD holds the offsets to at least one sub-IFD which contains the raw image data. Some cameras also have an additional sub-IFD which contains a JPEG version of the image.

The direntry with tag 0x8769 "EXIF" of the main IFD holds the offset to the EXIF IFD, which contains EXIF metadata. EXIF tags are well documented.

The direntry with tag 0x927c "MakerNote" of the EXIF IFD hold the offset to the makernote, which is a data structure containing more meta-data as well as the compression curves.

MakerNote

Rather than adding another (sub-)IFD as the makernote, Nikon instead decided to embed another full TIFF document.

At the makernote offset, you should find the null-terminated magic string "Nikon". After that come two 64 bit that might contain a version tag.

Afterwards, you should find another TIFF magic string. Careful: Since the endianness of a TIFF document depends on the magic ("MM" vs "II"), it is theoretically possible that this sub-TIFF has a different one than the outer TIFF, however I have not yet encountered such cursed files.

The makernote should be the main IFD of the sub-TIFF. All offsets inside it are relative to the sub-TIFF instead of global file offsets.

The direntry with tag 0x8c "ContrastCurve" in the makernote gives the offset to the contrast curve, the one with tag 0x96 "LinearsiationTable" the offset to the images compression curve.

Linearisation Table

At the linearisation table offset, you can read two unsigned bytes, henceforth referred to as version0 and version1.

According to dcraw, if version0 equals 0x49 you need to seek forward by 2110 now.

Now you can read four 16 bit unsigned integers, the vpred table, and another 16 bit unsigned integer, the curve size.

There are multiple different curve types with different decoding strategies.

After decoding the table, it may have a "tail" of repeating values, which needs to be pruned.

Type A

Calculate the maximum amount of elements in the curve using the bits per sample, which are either 12 or 14. TODO show how to get bps! 

const max = 1 << bps;

Calculate the step size. 

const step = max / (curve_size - 1);

Read as many unsigned 16 bit integers as indicated by the curve size, spaced out by the step. 

for (0..curve_size) |i|
    curve[i * step] = takeInt(u16);

To fill in the elements in between the values you have just read from the file, calculate the average of the two nearest values from the file, weighted by index distance. 

for (0..max) |i|
    curve[i] = (curve[i - i % step] * (step - i % step) + curve[i - i % step + step] * (i % step)) / step;

TODO XXX trim

Type B

This curve type needs no decoding. Simply read as many unsigned 16 bit integers as the curve size.

TODO XXX trim